python - How to accumulate unique sum of columns across pandas index -


i have pandas dateframe, df created with

df = pd.read_table('sorted_df_changes.txt', index_col=0, parse_dates=true, names=['date', 'rev_id', 'score'])

which structured so:

                     page_id     score   date 2001-05-23 19:50:14  2430        7.632989 2001-05-25 11:53:55  1814033     18.946234 2001-05-27 17:36:37  2115        3.398154 2001-08-04 21:00:51  311         19.386016 2001-08-04 21:07:42  314         14.886722

date index , of type datetimeindex.

every page_id may appear in 1 or more dates (not unique) , large in size ~1 million. of pages make document.

i need score entire document @ every time in date while counting latest score given page_id.

example

example data

                     page_id     score   date 2001-05-23 19:50:14  1           3 2001-05-25 11:53:55  2           4 2001-05-27 17:36:37  1           5 2001-05-28 19:36:37  1           1

example solution

                     score   date 2001-05-23 19:50:14  3 2001-05-25 11:53:55  7 (3 + 4) 2001-05-27 17:36:37  9 (5 + 4) 2001-05-28 19:36:37  5 (1 + 4)

the entry 2 counted continuously since not repeated each time id 1 repeated new score replaces old score.

edit:

finally, found solution don't need loop:

df.score.groupby(df.page_id).transform(lambda s:s.diff().combine_first(s)).cumsum()

i think loop needed:

from stringio import stringio txt = """date,page_id,score 2001-05-23 19:50:14,  1,3 2001-05-25 11:53:55,  2,4 2001-05-27 17:36:37,  1,5 2001-05-28 19:36:37,  1,1 2001-05-28 19:36:38,  3,6 2001-05-28 19:36:39,  3,9 """  df = pd.read_csv(stringio(txt), index_col=0)  def score_sum_py(page_id, scores):     itertools import izip     score_sum = 0     last_score = [0]*(np.max(page_id)+1)     result = np.empty_like(scores)     i, (pid, score) in enumerate(izip(page_id, scores)):         score_sum = score_sum - last_score[pid] + score         last_score[pid] = score         result[i] = score_sum     result.name = "score_sum"     return result  print score_sum_py(pd.factorize(df.page_id)[0], df.score)

output:

date 2001-05-23 19:50:14     3 2001-05-25 11:53:55     7 2001-05-27 17:36:37     9 2001-05-28 19:36:37     5 2001-05-28 19:36:38    11 2001-05-28 19:36:39    14 name: score_sum

if loop in python slow, can try convert 2 series page_id, scores python list first, loop on list , calculation python's native integer maybe faster.

if speed important, can try cython:

%%cython cimport cython cimport numpy np import numpy np  @cython.wraparound(false)  @cython.boundscheck(false) def score_sum(np.ndarray[int] page_id, np.ndarray[long long] scores):     cdef int     cdef long long score_sum, pid, score     cdef np.ndarray[long long] last_score, result      score_sum = 0     last_score = np.zeros(np.max(page_id)+1, dtype=np.int64)     result = np.empty_like(scores)      in range(len(page_id)):         pid = page_id[i]         score = scores[i]         score_sum = score_sum - last_score[pid] + score         last_score[pid] = score         result[i] = score_sum      result.name = "score_sum"     return result

here use pandas.factorize() convert page_id array in range 0 , n. n unique count of elements in page_id. can use dict cache last_score of every page_id without using pandas.factorize().


Comments

Post a Comment

Popular posts from this blog

monitor web browser programmatically in Android? -

i've got tricky question here. need users make payment bank (namely barclaycard) in uk. so, have https url , add parameters (such amount pay, order reference, etc) url, start http connection intent.actionview, redirect user browser can enter credit card details on bank's webpage , make payment our account successfully. far ? the code use below (i changed values privacy reasons) problem is, need app when user has completed/failed/cancelled payment. barclaycardautomatically redirects particular url when payment has succeeded, 1 if failed. there no way of knowing when barclaycard payment has succeeded go android app somehow ? button cardbutton = (button) findviewbyid(r.id.card_button); cardbutton.setonclicklistener(new view.onclicklistener() { @override public void onclick(view arg0) { string prehashstring = new string(); string prohashstring = new string(); string shapassphrase = new string(); shapassphrase = "gsvth£h70zkh...
Read more

Shrink a YouTube video to responsive width -

i have youtube video embedded on our website , when shrink screen tablet or phone sizes stops shrinking @ around 560px in width. standard youtube videos or there can add code make go smaller? you can make youtube videos responsive css. wrap iframe in div class of "videowrapper" , apply following styles: .videowrapper { float: none; clear: both; width: 100%; position: relative; padding-bottom: 56.25%; padding-top: 25px; height: 0; } .videowrapper iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } the .videowrapper div should inside responsive element. padding on .videowrapper necessary keep video collapsing. may have tweak numbers depending upon layout.
Read more

wpf - PdfWriter.GetInstance throws System.NullReferenceException -

i'm trying create pdf document using itextsharp 5.3.4 using following document document = new document(); filestream stm = new filestream(filename, filemode.create); pdfwriter writer = pdfwriter.getinstance(document, stm); i'm getting system.nullreferenceexception following stack trace: system.nullreferenceexception occurred hresult=-2147467261 message=la référence d'objet n'est pas définie à une instance d'un objet. source=itextsharp stacktrace: à itextsharp.text.version.getinstance() innerexception: i've verified neither document nor stm null, , if select "continue" in vs12 document created - exception thrown. updated itextsharp 5.4.0 , it's still occurring. can't find information on anywhere - got ideas? make sure not catching exceptions. nullreferenceexception can 1 caught , handled inside itextsharp, don't care it. fact can continue supports theory. change following setting verify: debug -> exceptions ...
Read more