Is java.util.Hashtable more memory efficient than containers.Map / struct?

Erik Johannes Loo

2022 3 2

0 个回答

5 次查看（30 天）

0 个投票

I am saving large amounts of traffic data in a format suitable for real-time data analysis/visualisation. I need quick/constant access to traffic data corresponding to a given day/direction. At the moment, I am structuring my data as follows:

containers.Map containing one key per direction+date. The value is a struct with fields Speed (single array), Flow (single array), Incidents (array), etc.

The issue is that the files easily exceed 1GB, so I am trying to minimise their footprint.

I did a quick test storing 2 arrays of 100x100 uint8 in different formats and found java.util.Hashtable to be surprisingly small:

- array 20kB

- struct 29kB

- cell 24kB

- map 24kB

- java hashtable 8kB

I am considering to swap my containers.Map and structs with nested java hashTables. Is this a good idea? And how come the size of the file is so small for a java.Hashtable? (I am assuming some kind of compression is happening there)

3 个评论
显示 1更早的评论隐藏 1更早的评论

Erik Johannes Loo 2022-3-2

There is definitely something interesting happening. I used arrays of zeros in the first test, and i've now used one 100x100 single rand arrays, but the results are still consistent:

- array 40kB

- struct 43kB

- cell 42kB

- map 54kB

- java hashtable 42kB

I can understand struct/cell seem to have a 2/3kB overhead per fieldname/index. What's surprising about the java hashtable is that adding more fields does not seem to increase its overhead as with structs/cells/maps.

If I save 2 arrays, the size is 78kB. If I save a java hash table with 1 field being another java hashtable with an array and the second field being just the array the size is also 78kB. If both fields have the same array, the size is 44kB.

Walter Roberson 2022-3-2

Each place that can contain data of different size or type has a header of about 108 bytes (I think I found 104 bytes in one case.) Each field of each entry of a struct array has a different size / type so it can add up. Likewise each cell entry has the same overhead.

I seem to recall that I measured the storage space for fieldnames, but I do not clearly recall what I found. Possibly 64 bytes per field per struct array. (Struct arrays have the same organization for each entry so the field names do not need to be repeated.)

请先登录，再进行评论。

请先登录，再回答此问题。

Follow Question