How to adjust append() to concat()?

Python Help
1

Hi, I am very new to python and attempting to replicate this code. After some researching I found that append() is no longer in the pandas library and now you have to use concat(). I am trying to run the code below and wondering how I could adjust it? Any help or advice would be greatly appreciated!

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, mean_squared_error
from sklearn.impute import SimpleImputer

df = df.sort_values(by='Date', ascending=True)

forecast_horizons = [1, 7, 14, 21, 30]

results = pd.DataFrame(columns=['Horizon', 'RMSE', 'MAPE'])

for forecast_horizon in forecast_horizons:
    for i in range(1, 13):
        df[f'lag_{i}'] = df['forecast_vol'].shift(i)
    train_data = df.iloc[:-forecast_horizon, :]
    test_data = df.iloc[-forecast_horizon:, :]

    X_train = train_data.drop('forecast_vol', axis=1)
    y_train = train_data['forecast_vol']
    X_test = test_data.drop('forecast_vol', axis=1)
    y_test = test_data['forecast_vol']

    imputer = SimpleImputer(strategy='mean')

    imputer.fit(X_train)

    X_train = imputer.transform(X_train)
    X_test = imputer.transform(X_test)

    model = Ridge(alpha=1, random_state=42)
    model.fit(X_train, y_train)

    X_forecast = df.drop('forecast_vol', axis=1).tail(forecast_horizon)
    y_forecast = model.predict(X_forecast)

    rmse = np.sqrt(mean_squared_error(y_test, y_forecast))

    mape = np.mean(np.abs((y_test - y_forecast) / y_test)) * 100

    results = results.append({'Horizon': forecast_horizon, 'RMSE': rmse, 'MAPE': mape}, ignore_index=True)

print(results)
2

I get this error when attempting to run AttributeError: ‘DataFrame’ object has no attribute ‘append’

3

Hello,

can you type your code as per these guidelines:

format_code
format_code454×506 32.1 KB

This way, others can copy your code for testing so as not having to retype your script from scratch (help us help you).

4

Thank you! I will paste code in correct format.

5

I have made the change, hopefully I have pasted all that is needed to get help, if I havent or there is more information you need please let me know, thank you

6

Hi,

initial feedback:

Is this work related that you have to figure this out or are you using this as an exercise to learn Python?

The reason that I am asking is because there is a lot to unpack here. Do you have previous programming experience - maybe with another language, and know programming fundamentals? Are you already familiar with Pandas library package?

7

this is related to my university dissertation and i want to see if i am able to run this code from a paper i found my self. so abit of both trying to figure this out and using it to learn more about python

8

Ok, got it.

Well, for starters, if you have yet to have created a variable, you cannot use or reference it. In the following line:

df = df.sort_values(by='Date', ascending=True)

Where was df initially defined / assigned that you are now referencing it? Notice that it is being referenced many times throughout. If you first don’t create it, you will not be able to go any further.

9

By the way, you don’t have to append the data, you can simply add the data as per this simple example:


import pandas as pd

# Create emply dataframe with only column titles (unnecessary, can do
# together at the end when data is available)
results = pd.DataFrame(columns=['Horizon', 'RMSE', 'MAPE'])
print('The dataframe with no data:')
print(results)

# Raw data - assumed data obtained after the iteration processing loops
data1 = [1,2,3,4,5]
data2 = [6,7,8,9,10]
data3 = [11,12, 13, 14, 15]

# Update dataframe with column titles and raw data
updated_results = pd.DataFrame(({'Horizon': data1,
                    'RMSE': data2,
                    'MAPE': data3}))

print('\nThe new updated dataframe is:')
print(updated_results)

I noticed that you initially created an empty dataframe by only creating it with the column titles. You can do this after the data has been processed and create the dataframe after you have the required data.

I would recommend looking up some additional examples online so that you get more familiar with the pandas library package and Python in general.

10

Just seen the full message, i have experience in VBA and know very basic fundamentals and not familiar with Pandas library, I know the code is complex especially for a beginner like me but I wanna dive into the deep end on this one ahaha. I should have said but my df was defined earlier on in the code. So I updated the code and moved the creation of the empty and updated dataframe to after the required data but now i get a ValueError: If using all scalar values, you must pass an index

for forecast_horizon in forecast_horizons:
    for i in range(1, 13):
        df[f'lag_{i}'] = df['forecast_vol'].shift(i)
    train_data = df.iloc[:-forecast_horizon, :]
    test_data = df.iloc[-forecast_horizon:, :]

    X_train = train_data.drop('forecast_vol', axis=1)
    y_train = train_data['forecast_vol']
    X_test = test_data.drop('forecast_vol', axis=1)
    y_test = test_data['forecast_vol']

    imputer = SimpleImputer(strategy='mean')

    imputer.fit(X_train)

    X_train = imputer.transform(X_train)
    X_test = imputer.transform(X_test)

    model = Ridge(alpha=1, random_state=42)
    model.fit(X_train, y_train)

    X_forecast = df.drop('forecast_vol', axis=1).tail(forecast_horizon)
    y_forecast = model.predict(X_forecast)

    rmse = np.sqrt(mean_squared_error(y_test, y_forecast))

    mape = np.mean(np.abs((y_test - y_forecast) / y_test)) * 100

results = pd.DataFrame(columns=['Horizon', 'RMSE', 'MAPE'])
updated_results = pd.DataFrame(({'Horizon': forecast_horizon,'RMSE': rmse,'MAPE': mape}))
print(updated_results)
11

You no longer need this line:

… since you will be making use of this one (as per instructions in previous script example):

Which line does this pertain to?

One small suggestion as you’re developing/testing/verifying your script, start small. Add one line, test it (print the results - see if it matches your expected results). If it satisfies the expected results, then add the next line. If it doesn’t meet expected results, backtrack and figure out why. Generally, don’t develop a script and wait to test till the very end. You should be testing/verifying as you’re developing your script.

2 Likes
12

Okay understood, thank you very much for your help! I really appreciate it! The whole script is long so I have been attempting to recreate it line by line but in this scenario I wasn’t sure what was causing the issue so pasted a section.

ValueError: If using all scalar values, you must pass an index

This error pertains to this line below

updated_results = pd.DataFrame(({'Horizon': forecast_horizon,'RMSE': rmse,'MAPE': mape}))
13

Remember, the print statement can be your best friend.

Any value that you wish to verify, print it. Since the line above contains three independent variables, print them to verify their contents. If all appears good, then go up the script, line by line until you get to the root cause of the exception. If any of them has a discrepancy, backtrack and find out why.

print(forecast_horizon)
print(rmse)
print(mape)