Sweet-toothed latte or flat white drinkers drink the same amount of calories as three-and-a half cans of Coke, nutritionists have said.
Adding sugar, flavoured syrups, cream, whole milk or semi-skimmed to their weekly grind means they take in over 480 more needless calories than black unsweetened coffee drinkers.
Tea drinkers, generally, were healthier but those who took sugar, honey or milk to make their brew more palatable drank over 300 more calories than those who took their daily cuppa plain - the equivalent to over two cans of Coke.
As most took just a dash of milk, over 85 per cent of these calories came from sugar.
One 330ml can of Coca Cola has 139 calories.
While a little amount of sugar, dash of syrup or cream may not seem much, they all add up and help pile on the pounds, University of Illinois scientists said.
Instead they should avoid the hazelnut or vanilla syrup and whipped cream, and go back to basics by having it plain with low fat or fat free milk, sweeteners or just black.
And when getting a coffee from a coffee chain, avoid supersized coffees and order the smallest.
The study looked at the drinking habits of adults Americans and found over half drink coffee and over a quarter drink tea on any given day.
This equates to more than 160 million Americans.
But roughly two-thirds of the coffee drinkers and one-third of the tea drinkers put sugar, cream, flavourings or other calorie-rich additives in their drinks.
Professor Ruopeng An said: "Coffee and tea are among the most widely consumed beverages in US adults.
"Unlike other popular beverages including alcohol and sugar-sweetened beverages that are typically consumed in isolation, many people prefer drinking coffee
and tea with add-ins like sugar or cream.
"These add-in items are often dense in energy and fat but low in nutritional value.
"Drinking coffee and tea with add-ins on a regular basis might impact an individual's daily energy/nutrient intake and diet quality.
"The 2015 to 2020 Dietary Guidelines for Americans suggests that 'coffee, tea, and flavored waters also can be selected, but calories from cream, added sugars, and other additions should be accounted for within the eating pattern.'
"To our knowledge, no study has been conducted to assess consumption of coffee and tea with add-ins in relation to daily energy and nutrient intake at the population level. "
He noted added milk may add a bit of calcium to the diet, but the amount - on average 22 milligrammes per day - was negligible.
The daily recommended calcium intake was 1,000 to 1,300 milligrams, depending on one's age and pregnancy status.
Black coffee drinkers had about 69 fewer total calories per day, on average, than those who add sweeteners, cream or other substances to their coffee.
More than three fifths of those calories came from sugar, with fat accounting for most of the rest of the extra calories consumed.
Prof An added tea drinkers tended to add fewer calorie-dense substances to their tea if they add anything at all.
He said: "Compared with adding nothing to one's tea, drinking tea with caloric add-ins increased daily caloric intake by more than 43 calories, on average, with nearly 85 per cent of those added calories coming from sugar.
"The daily intakes may seem small, but the extra calories every day can add up to extra pounds.
"Our findings indicate that a lot of coffee and tea drinkers regularly use caloric add-ins to improve the flavour of their beverages, but possibly without fully realising or taking into consideration its caloric and nutritional implications."
The study published in the journal Public Health looked at 12 years of data up to 2012 from the National Health and Nutrition Examination Survey.