Custom compressioncodec on Spark -


has developed java-based custom compressioncodec apache spark (i.e. org.apache.spark.io.compressioncodec)?

unlike older versions, requires implementation of 2 methods in class: compressedinputstream , compressedoutputstream

i'm working on this, , works smaller data sizes, shows "executor lost" errors larger sizes.

i'm seeing end() method of compressor class never called. wonder if might bug in spark lower level library (i'm using spark 1.5.1).

has been down path?

thanks.


Comments

Post a Comment

Popular posts from this blog

sql - can we replace full join with union of left and right join? why not? -

can replace full join union of left , right join? if no,why? 'yes' if t1 , t2 sets (no duplicated rows), otherwise answer 'no'. create table t1 (i int); create table t2 (i int); insert t1 values (1); insert t1 values (2); insert t1 values (2); insert t2 values (3); full join select * t1 full join t2 on t1.i=t2.i order 1,2 1 (null) 2 2 2 2 (null) 3 union select * t1 left join t2 on t1.i=t2.i union select * t1 right join t2 on t1.i=t2.i order 1,2 1 (null) 2 2 (null) 3 union all select * t1 left join t2 on t1.i=t2.i union select * t1 right join t2 on t1.i=t2.i order 1,2 1 (null) 2 2 2 2 2 2 2 2 (null) 3
Read more

javascript - Parallax scrolling and fixed footer code causing width issues -

i bolted bits of code. 1 parallax scrolling, , other piece of fixed footer code found here: http://codepen.io/madshaakansson/pen/cgjch though after bolting , fixing few issues, have problem different widths. header different width footer, , main content doesnt span whole width of page. have tried adjusting width of content css 100%, made main content span whole width, ended breaking footer. i not sure need change make these different bits of code work together. advice appreciated. my attempt: http://codepen.io/dingerzat/pen/qgbwkk html /*global define: false */ function init() { window.addeventlistener('scroll', function(e) { var distancey = window.pageyoffset || document.documentelement.scrolltop, shrinkon = 300, header = document.queryselector("header"); if (distancey > shrinkon) { classie.add(header, "smaller"); } else { if (classie.has(header, ...
Read more

iOS: Performance of reloading UIImage(name:...) -

i have tableview , each row has icon. 90% of rows have same icon. in cellforrowatindexpath method load image assets folder this: var cellimage = uiimage(named: "myimage") does pose performance problem? os somehow smart enough it's same image each time? better if loaded image once , saved in class variable , assigned each time? uiimage 's named: api does kind of caching. if @ uiimage's documentation , states: use init(named:in:compatiblewith:) method (or init(named:) method) create image image asset or image file located in app’s main bundle (or other known bundle). because these methods cache image data automatically, recommended images use frequently.
Read more