What are the main differences between DXF and DWG formats? And why is my DXF drawing 5 times bigger than the same drawing in DWG format? And out of curiousity what are big contributors to drawing sizes?

You can look at the DWG and DXF AutoCAD formats similar to the DOC and RTF MSWord formats. The DWG (and DOC) is the native (binary) format while DXF (and RTF) are primarly textual file exchange formats (with an open, documented structure).
 
The textual format of DXF is the main reason of size differences.