python - Caching CSV-read data with pandas for multiple runs -


i'm trying apply machine learning (python scikit-learn) large data stored in csv file 2.2 gigabytes.

as partially empirical process need run script numerous times results in pandas.read_csv() function being called on , on again , takes lot of time.

obviously, time consuming guess there must way make process of reading data faster - storing in different format or caching in way.

code example in solution great!

i store parsed dfs in 1 of following formats:

all of them fast

ps it's important know kind of data (what dtypes) going store, because might affect speed dramatically


Comments

Post a Comment

Popular posts from this blog

How to get the arclen between two curve points in Maya? -

in maya 2015, can arclen of curve using command: cmds.arclen('bezier1') but want arclen of 2 points in curve. there anyway this? using maya's api surely best way @scottiedoo said, here function made when didn't know api , gives same results. from maya import cmds def computecrvlength(crv, startparam = none, endparam = none): ''' compute length of curve between 2 given uparameters. if both uparameters arguments set none (default), compute length of whole curve. arguments: - crv = string; existing nurbcurve - startparam = 0 <= float <= 1 or none; default = none; point parameter value, if not none, compute points between startpt , endpt values. - endparam = 0 <= float <= 1 or none; default = none; point parameter value, if not none, compute points between startpt , endpt values. returns: - length of curve between given uparameters - length of curve start startp...
Read more

C++ Linked List -

i trying create linked list download txt file , use linked list handle file line line. when handling downloaded linked list operations performed on such text editor would. encountering problems however. seems "node(string value)" section of code has wrong though original node() declaration no arguments passes. unable figure out quite is. node.h class node { public: node(); node(string value); void setnext(node *nextnode); // allows user set "next" pointer of node points friend class linkedlist; private: string data; // data box node* next; // pointer box }; node.cpp # include <string> # include "node.h" using namespace std; node::node() { data = ""; next = null; } node::node(string value) { data = value; next = null; } void node::setnext(node *nextnode) // allows user set "next" pointer of node points { this->next = nextnode; } your #include <string> ...
Read more

git svn - Unignore directories in git-svn (adding directories to a sparse checkout) -

when using git svn, sparse checkout of central repository may achieved along lines of git clone -s url-to-root --include-paths 'dir1|dir2|dir3' after using setup time, add directory dir4 tracked. directory has existed in many commits have checked out. editing variable svn-remote.svn.include-paths running git config --edit does not have desired effect on git fetch (only files changed in future commits checked out changed; not immediately), , following not check out dir4: git svn fetch --include-paths dir4 i think following work ineffective due refetching revisions since first revision @ dir4 appeared first time (let's call rev): git svn reset -r rev git svn fetch --include-paths dir4 git svn rebase what better way of tracking dir4? if not care history of dir4? is there better way initial sparse checkout git-svn if suspect want change list of tracked directories later? have tried starting listing directoreis not fetched in original commit, not know how...
Read more