python - How to manage data batches for big neural network? -


i preparing learn quite big neural network (fc, conv, pool, etc.) millions of small images ( ~~100x100 px, 3 channels each) in keras. files around ~~800 gb , there question. how should perpare data?

i know keras handles batches better learn network either 100 files 8 gb each or create ~~300k files (in each merged 32 or 64 images)? think better have bigger files , faster read them 8 times (8 big files) 300k times not sure.

i have got less 100 gb ram can not load whole data @ once sure.

thanks!

you can use keras.preprocessing.image.imagedatagenerator provided keras instead of loading files memory. allows set batch size. imagedatagenerator can augment data in real-time "for free" if need. since takes time train network using batch of images, reading files hard drive doesn't slow down performance. main bottleneck computational power.

the interface , examples of keras.preprocessing.image.imagedatagenerator can found @ imagedatagenerator


Comments

Post a Comment

Popular posts from this blog

php - How to add and update images or image url in Volusion using Volusion API -

actually have store importing products store product data insert , update image or image url not insert , update. this xml code name datapro.txt. <products_joined> <productcode>3710_012t</productcode> <vendor_partno>eah5450silent/di/1gd3(lp)</vendor_partno> <productname>test product ta</productname> <hideproduct>n</hideproduct> <stockstatus>20</stockstatus> <lastmodified>1/5/2016 10:25:00 am</lastmodified> <lastmodby>2</lastmodby> <productweight>0.9</productweight> <productprice>100</productprice> <productmanufacturer>asus tek</productmanufacturer> <vendor_price>32.69</vendor_price> <numproductssharingstock>0</numproductssharingstock> <categoryids>107</categoryids> <producturl>http://tebkq.mvlce.servertrust.com/productdetails.asp?productcode=3710_012t</producturl>
Read more

javascript - IE9 error '$'is not defined -

Image
<!--[if lte ie 9]><html class="lte9"><![endif]--><!--[if gt ie 9]><html><![endif]--> <!--[if !ie]><!--> <html> <!--<![endif]--> blah blah <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.1/jquery.min.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script> <script src="{{ asset('js/mypage.js') }}" type="text/javascript"></script> </body> </html> when load page, ie9 brings error '$' not defined . how can solve error? +++ mypage.js $(function(){ //when load page, //ie9 tag not moving. $(window).load(function() { $('div.mask').hide(); }); // when click , willshow show. //for not ie 9 if (!$('html').hasclass('lte9')) { var = $('ul.about > li > span.activebtn');
Read more