Brown was drafted by the Pats out of Marshall University in the 8th round of the 1993 NFL Draft. He was waived as a final cut in the 1994 preseason but was resigned on October 19. For the first few years of his career, he was considered a solid third or fourth receiver and an excellent special teamer. He returned punts for the Patriots in the 1993-4 and 1994-5 seasons. In 1997, he recorded 41 catches for 607 yards and 6 TDs despite being behind both Terry Glenn and Shawn Jefferson on the depth chart at receiver as well as competing with Ben Coates for catches. In 1998, he resumed his duties as a punt returner.
His first year as a starter was 2000, and in 2001 he, alongside Tom Brady, led the Patriots to their first ever Super Bowl championship, recording 83 catches during the season for 944 yards. During the AFC Championship game against the Steelers that season, Brown returned a crucial punt for a touchdown which provided the winning margin, adding to the two he returned for TDs during the regular season. In 2002, he recorded 101 receptions for 1199 yards and 5 TDs, good for fifth in the NFL. In 2003, he again lead the Patriots in receiving with 97 catches for 890 yards and helped lead the team back to the Super Bowl.
In 2004, he had only 17 receptions, but gave an enormous contribution in what was originally an emergency role on defense, ranking second on the team in interceptions with three. He was topped in this category only by Eugene Wilson. In the Pats 20-3 playoff victory over the Colts, he completely shut down receiver Brandon Stokley.
Brown was released by the Patriots on March 1, 2005 for salary cap reasons, but he signed a new contract with them on May 23, 2005. He signed despite a better financial deal from the New Orleans Saints; the income he receives from Boston-area commercial endorsement deals makes up much of the difference. In the 2005-6 season, he recorded 39 receptions for 466 yards. In the AFC Divisional playoff game against the Denver Broncos, Brown uncharacteristically muffed a punt that would end the Patriots' hope for a comeback.
In addition to playing offense and defense with success, Brown is the Patriots all-time leading punt returner with 244 returns for 2,554 yards and 3 TDs. He is second all-time in Patriots history in both receptions (514) and receiving yards (5982).