# Python for data science: Parsing "aTimeLogger" Android app data to graphs using pandas and matplotlib

**URL:** <https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612>\
**Category:** Python Help\
**Created:** [June 12, 2024, 7:32pm UTC](https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612 "2024-06-12T19:32:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![enoren5](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/enoren5/32/5568_2.png) [@enoren5](https://discuss.python.org/u/enoren5)\
**Post date:** [June 12, 2024, 7:32pm UTC](https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612/1 "2024-06-12T19:32:25Z")

</div>

Greetings Pythonistas!

I’ve got a data set in CSV format ([available for download here](https://www.dropbox.com/scl/fi/ubxm2ofjc4lufqq4vcnvq/all-comments-removed.csv?rlkey=cm64jttj1t6da41gy0ictt6p6&st=zbay3wno&dl=0)) which spans from July 2019 up to December 2023. It is a collection of activities (and their time intervals) that I’ve meticulously recorded and tracked while doing social science research like Philosophy as well as learning how to program Python and build websites with Django, among other research activities.

To give you a specific example about how I use this tool, if I were to, for example, spend 25 minutes on my lunch hour on my tablet watching Udemy course content teaching the basics of Binary Search Trees, at the end of my lunch break, using an app called [aTimeLogger](http://www.atimelogger.com/) on my Android phone I enter the “Activity” type such as “algorithms”. Next I enter metadata such the date, the start time, and the end time. Then I write an annotation (1-2 sentences) which is sort of like a mental note for my future reference. Other discrete research activity categories could be “Python”, “Django”, “sysadmin”, or even “Magick” / “writing”).

For additional context, over the range of the data set, I’ve spent a total of ~378 hours doing something “Python” related and ~579 hours working on Django course content (or a Django based web project). Those are the two largest categories. The rest of the Activities aren’t as data dense.

My two latest code snippets and graphs can be found here.

Here are my questions:

1. In the first snippet and graph below, in my Jupyter Notebook pandas and matplotlib show two categories - - but only partially so. I noticed that when I change the alpha (translucency) variable, the time spent on the different categories overlap each other. How do I stack the data instead? That’s my first question.

```python
import pandas as pd
pd.set_option('display.expand_frame_repr', False)
import matplotlib.pyplot as plt
  
bulk_df = pd.read_csv('data/all-comments-removed.csv', parse_dates=["From", "To"])
bulk_df['Duration'] = pd.to_timedelta(bulk_df['Duration'])
bulk_df['Duration_hours'] = bulk_df['Duration'].dt.total_seconds() / 3600
 
# Copy so changes made to python_df dos not affect bulk_df and vice versa
python_df = bulk_df[bulk_df["Activity"] == "Python"].copy()
python_df.set_index('From', inplace=True)
 
# Calculate rolling means using the index now
python_df['Rolling_Mean_90'] = python_df['Duration_hours'].rolling('90D').mean()
python_df['Rolling_Mean_182'] = python_df['Duration_hours'].rolling('182D').mean()
 
# Copy so changes made to django_df dos not affect bulk_df and vice versa
django_df = bulk_df[bulk_df["Activity"] == "Django"].copy()
django_df.set_index('From', inplace=True)
# Calculate rolling means using the index now
django_df['Rolling_Mean_90'] = django_df['Duration_hours'].rolling('90D').mean()
django_df['Rolling_Mean_182'] = django_df['Duration_hours'].rolling('182D').mean()
 
python_df_Month = python_df['Rolling_Mean_90'].resample('MS').sum()
django_df_Month = django_df['Rolling_Mean_90'].resample('MS').sum()
# py_dj_Month_combined = python_df_Month.add(django_df_Month, fill_value=0)
 
plt.figure(figsize=(14, 8))
plt.bar(python_df_Month.index, python_df_Month, label='Python 90-Day Rolling Mean',width=20, alpha=0.5) # color='red')
plt.bar(django_df_Month.index, django_df_Month, label='Django 90-Day Rolling Mean', width=20, alpha=0.5) #, color='blue')
plt.legend()
plt.title('Stacked Bar Chart for Python and Django Activities')
plt.xlabel('Date')
plt.ylabel('Hours Spent')
plt.show()

```

That shows as:

 ![Screenshot from 2024-06-12 14-48-59](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/0/a/0a4e1d3a088ce08a8a520345139febee81d06dbb.png)

1. In the second snippet and graph below, only one category shows up (”Magick”). How do I get the other “Research” category to show? As far as I can tell, the way I handle the data and cast function calls and methods against the two dataframes should work. I’ve been swapping out variable names, tried refactoring, as well as making large and small other changes without success. Who here can identify what I might be missing to get both categories to show (instead of one)? I feel like what I am missing is trivial and obvious. I am hoping another forum member here can point out the blatent error I am making. (My additional intent here is to ensure the bar graph also stacks the data (rather than overlapping) like I have set out to do with the first graph).

```python
import pandas as pd
pd.set_option('display.expand_frame_repr', False)
import matplotlib.pyplot as plt
 
# Load the data
bulk_df = pd.read_csv('data/all-comments-removed.csv', parse_dates=["From", "To"])
bulk_df['Duration'] = pd.to_timedelta(bulk_df['Duration'])
bulk_df['Duration_hours'] = bulk_df['Duration'].dt.total_seconds() / 3600
 
# Copy and filter data for "Magick" activity and calculate rolling means
magick_df = bulk_df[bulk_df["Activity"] == "Magick"].copy()
magick_df.set_index('From', inplace=True)
magick_df['Rolling_Mean_90'] = magick_df['Duration_hours'].rolling('90D').mean()
magick_df['Rolling_Mean_182'] = magick_df['Duration_hours'].rolling('182D').mean()
 
# Copy and filter data for "Research (general)" activity and calculate rolling means
research_df = bulk_df[bulk_df["Activity"] == "Research (general)"].copy()
research_df.set_index('From', inplace=True)
research_df['Rolling_Mean_90'] = research_df['Duration_hours'].rolling('90D').mean()
research_df['Rolling_Mean_182'] = research_df['Duration_hours'].rolling('182D').mean()
 
# Resample data
magick_df_Month = magick_df['Rolling_Mean_90'].resample('MS').sum()
research_df_Month = research_df['Rolling_Mean_90'].resample('MS').sum()
 
# Plot the combined data with wider bars
plt.figure(figsize=(12, 6))
plt.bar(research_df_Month.index, research_df_Month, label='"Research" 90-Day Rolling Mean', width=20, alpha=0.5, color='blue')
plt.bar(magick_df_Month.index, magick_df_Month, label='"Magick" ("Philosophy") 90-Day Rolling Mean',width=20, alpha=0.5, color='red')
 
plt.legend()
plt.title('Stacked Bar Chart for Magick and Research Activities')
plt.xlabel('Date')
plt.ylabel('Hours Spent')
plt.show()

```

 ![Screenshot from 2024-06-12 14-50-16](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/d/2/d232e12bd27f0332c46e26b45b9dee417adfb555.png)

---

<div class="post-metadata">

**Author:** ![cameron](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cameron/32/2658_2.png) [@cameron](https://discuss.python.org/u/cameron)\
**Post date:** [June 12, 2024, 11:10pm UTC](https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612/2 "2024-06-12T23:10:26Z")

</div>

If you get no good answer here, try asking over in the matplotlib  
Discourse: [Community - Matplotlib](https://discourse.matplotlib.org/c/community/11)

> 1. In the first snippet and graph below, in my Jupyter Notebook pandas  
> and matplotlib show two categories - - but only partially so. I noticed  
> that when I change the alpha (translucency) variable, the time spent on  
> the different categories overlap each other. How do I stack the data  
> instead? That’s my first question.

Have a look at this example:  
[https://matplotlib.org/stable/gallery/lines\_bars\_and\_markers/bar\_stacked.html](https://matplotlib.org/stable/gallery/lines_bars_and_markers/bar_stacked.html)

I found this in the docs for `pyplot.bar` where it says:  
“Stacked bars can be achieved by passing individual bottom values per  
bar. See Stacked bar chart.”  
and links to [matplotlib.pyplot.bar — Matplotlib 3.9.0 documentation](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.bar.html)

> 1. In the second snippet and graph below, only one category shows up  
> (”Magick”). How do I get the other “Research” category to show? As far  
> as I can tell, the way I handle the data and cast function calls and  
> methods against the two dataframes should work. I’ve been swapping out  
> variable names, tried refactoring, as well as making large and small  
> other changes without success. Who here can identify what I might be  
> missing to get both categories to show (instead of one)?

Is it possible that the Research category is (a) overlaid by the Magick  
category, hiding it or (b) entirely of size 0 or close to 0, making the  
bars invisibly small?

I’d try commenting out the Magick `.bar()` call and seeing what the  
graph looks like. if you get Research bars, pay particular attention to  
the scales on the axes.

---

<div class="post-metadata">

**Author:** ![enoren5](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/enoren5/32/5568_2.png) [@enoren5](https://discuss.python.org/u/enoren5)\
**Post date:** [June 16, 2024, 9:00am UTC](https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612/3 "2024-06-16T09:00:50Z")

</div>

> [@cameron](#):
>
> Is it possible that the Research category is (a) overlaid by the Magick  
> category, hiding it or (b) entirely of size 0 or close to 0, making the  
> bars invisibly small?

There is data:

 ![image](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/4/3/43ec8212f1d27da09546c36c56b77f9cd7c9890b.png)  
It’s not 0. Both the bar graphs have alpha set to 0.50 so I would be able to see both layered on top of each other.

I identified the problem: In my codesnippet above, I was referring to one of Series columns as “Research (general)”. When I changed it to just: “Research”, the data began plotting.

Here is my end product:

 ![image](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/1/c/1c327afdb825542ce120929726b87977d45b917b.png)

So that resolves my second question.

> [@cameron](#):
>
> Have a look at this example:  
> [Stacked bar chart — Matplotlib 3.9.0 documentation](https://matplotlib.org/stable/gallery/lines_bars_and_markers/bar_stacked.html)
> 
> I found this in the docs for `pyplot.bar` where it says:  
> “Stacked bars can be achieved by passing individual bottom values per  
> bar. See Stacked bar chart.”  
> and links to [matplotlib.pyplot.bar — Matplotlib 3.9.0 documentation](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.bar.html)

Thank you for sharing these doc links. At first glance, the variables and arguments used in the sample demos are very different from what I am working with. I’ll continue to experiment with refactoring my code to match what is described in these docs until I manage to successfully stack my bar charts.

If I encounter any more issues I will either report back here or reach out for additional support on the official matplotlib Discourse forum you suggested.

Thanks @cameron 🙂

---

<div class="post-metadata">

**Author:** ![cameron](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cameron/32/2658_2.png) [@cameron](https://discuss.python.org/u/cameron)\
**Post date:** [June 18, 2024, 4:47am UTC](https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612/4 "2024-06-18T04:47:08Z")

</div>

> [quote=“Cameron Simpson, post:2, topic:55612, username:cameron”]  
> Have a look at this example:  
> [Stacked bar chart — Matplotlib 3.9.0 documentation](https://matplotlib.org/stable/gallery/lines_bars_and_markers/bar_stacked.html)  
> […]  
> [/quote]
> 
> Thank you for sharing these doc links. At first glance, the variables and arguments used in the sample demos are very different from what I am working with. I’ll continue to experiment with refactoring my code to match what is described in these docs until I manage to successfully stack my bar charts.

The important bit is this code:

```
 bottom = np.zeros(3)

 for boolean, weight_count in weight_counts.items():
     p = ax.bar(species, weight_count, width, label=boolean, bottom=bottom)
     bottom += weight_count

```

The idea is to specify where the bottoms of the bars begin. It starts  
wit the array `bottom` all zeroes (the example only uses 3 bars). After  
plotting one set of bar it raises all the values in `bottom` by the size  
of the bar it just plotted. You can just add one array/series to  
another, eg:

```
 bottom += research_df_Month

```

and so forth.

You’d just need to add a `bottom=` parameter to your own `plt.bar()`  
calls and update it before the next bar plot call.

---

<div class="post-metadata">

**Author:** ![enoren5](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/enoren5/32/5568_2.png) [@enoren5](https://discuss.python.org/u/enoren5)\
**Post date:** [June 23, 2024, 7:09am UTC](https://discuss.python.org/t/python-for-data-science-parsing-atimelogger-android-app-data-to-graphs-using-pandas-and-matplotlib/55612/5 "2024-06-23T07:09:09Z")

</div>

Hi @cameron,

Thank you for your follow up. Based on the matplotlib doc, but especially the part you highlighted, I partially understand. Even though my end product didn’t resemble yours exactly, I still got it working.

For example, you suggested I use something like: `bottom += research_df_Month`. I didn’t use the addition assignment operator but I did end up adding one data frame object on top of the other for the `bottom` variable.

Here is my final graph. You’ll notice that I parsed 4 activities categories instead of just the 2 that I was working with previously. Below the image of my final graph, you can see my full working code snippet:

 ![image](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/d/8/d896621cac3ae4b8d8af835460bc4ee67d98cd7d.png)

```python
import pandas as pd
pd.set_option('display.expand_frame_repr', False)
import matplotlib.pyplot as plt

# Load the data
bulk_df = pd.read_csv('data/all-comments-removed.csv', parse_dates=["From", "To"])
bulk_df['Duration'] = pd.to_timedelta(bulk_df['Duration'])
bulk_df['Duration_hours'] = bulk_df['Duration'].dt.total_seconds() / 3600

# Copy and filter data for "Magick" activity and calculate rolling means
magick_df = bulk_df[bulk_df["Activity"] == "Magick"].copy()
magick_df.set_index('From', inplace=True)
magick_df['Rolling_Mean_90'] = magick_df['Duration_hours'].rolling('90D').mean()
magick_df['Rolling_Mean_182'] = magick_df['Duration_hours'].rolling('182D').mean()

# Copy and filter data for "Research (general)" activity and calculate rolling means
research_df = bulk_df[bulk_df["Activity"] == "Research"].copy()
research_df.set_index('From', inplace=True)
research_df['Rolling_Mean_90'] = research_df['Duration_hours'].rolling('90D').mean()
research_df['Rolling_Mean_182'] = research_df['Duration_hours'].rolling('182D').mean()

# Resample data
magick_df_Month = magick_df['Rolling_Mean_90'].resample('MS').sum()
research_df_Month = research_df['Rolling_Mean_90'].resample('MS').sum()

# Copy and filter data for "Python" activity and calculate rolling means
python_df = bulk_df[bulk_df["Activity"] == "Python"].copy()
python_df.set_index('From', inplace=True)
python_df['Rolling_Mean_90'] = python_df['Duration_hours'].rolling('90D').mean()
python_df['Rolling_Mean_182'] = python_df['Duration_hours'].rolling('182D').mean()

# Copy and filter data for "Django" activity and calculate rolling means
django_df = bulk_df[bulk_df["Activity"] == "Django"].copy()
django_df.set_index('From', inplace=True)
django_df['Rolling_Mean_90'] = django_df['Duration_hours'].rolling('90D').mean()
django_df['Rolling_Mean_182'] = django_df['Duration_hours'].rolling('182D').mean()

# Resample data
python_df_Month = python_df['Rolling_Mean_90'].resample('MS').sum()
django_df_Month = django_df['Rolling_Mean_90'].resample('MS').sum()

# Create a common index
common_index = magick_df_Month.index.union(research_df_Month.index).union(python_df_Month.index).union(django_df_Month.index)

# Reindex all Series to the common index
magick_df_Month = magick_df_Month.reindex(common_index, fill_value=0)
research_df_Month = research_df_Month.reindex(common_index, fill_value=0)
python_df_Month = python_df_Month.reindex(common_index, fill_value=0)
django_df_Month = django_df_Month.reindex(common_index, fill_value=0)

# Plot the combined data with stacked bars
plt.figure(figsize=(14, 8))

# Plot first activity
plt.bar(magick_df_Month.index, magick_df_Month, label='"Magick" ("Philosophy")', width=20, alpha=1, color='#3D9E60')

# Plot second activity, stacked on the first
plt.bar(research_df_Month.index, research_df_Month, label='"Research"', width=20, alpha=1, color='#2A6D20', bottom=magick_df_Month)

# Plot third activity, stacked on the first and second
plt.bar(django_df_Month.index, django_df_Month, label='"Django"', width=20, alpha=1, color='#FB947E', bottom=magick_df_Month + research_df_Month)

# Plot fourth activity, stacked on the first, second, and third
plt.bar(python_df_Month.index, python_df_Month, label='"Python"', width=20, alpha=1, color='#D95134', bottom=magick_df_Month + research_df_Month + django_df_Month)

plt.legend()
plt.title('Stacked Bar Chart for Python, Django, Magick, Research Activities 90-Day Rolling Mean')
plt.xlabel('Quarter (90 day increments)')
plt.ylabel('Hours Spent')
plt.show()

```