Project Meeting (30/1/2018)

Topics to Discuss

• What is the objective?

  • is it to find NN models better than ARIMA?
  • How to compare NN and ARIMA models - based on prediction accuracy or forecast accuracy(how long forecast)?

• Experimentation workflow

  • What data needs to be saved for model evaluation?

• NN input output size and how to recursively forecast

  • Does it make sense to have an NN that has smaller input than output and how to compute the recursive forecast?

• Train validation and test sets

  • Wilcoxon U test shows distributions of validation and test sets are different for some of the variables?

• Preprocessing

  • Is data seasonally adjusted?
  • Is it a good idea to remove NA rows so variables would have same length?
  • Fourier transform for seasonality detection/adjustment?

• Quarterly Time-Series Forecasting With Neural Networks Finds that simple NN models with 0 or 1 hidden nodes perform best on time series and in particular macroeconomic time series. They use a large data set of small series, of 50 observations each, and find that removing trend and seasonality improves results. The experiments in the paper follow the following procedure:

    For each data series
    	For each data transformation
    		Scale data to (-1,1)
    		For each model (form of input/output)
    			For NN architectures with hidden nodes
    				Train five NNs from random starting parameter
    				weights.
    				Keep the best of the five based on SSE
    			Scale the out-of-sample data using parameters
    			perform forecast on out-of-sample data
    			Unscale the forecasts
    			Untransform the forecasts using the appropriate
  

Proposed plan

• Write lit review section for feedforward NN
• Finally establish experimentation workflow
• Perform experiments and write them up
• Write lit review for recurrent NN
• Perform experiments with recurrent NN

Tasks Competed

• [x] Written evaluation metric subsection in report.

• [x] Implemented infrastructure for creation and evaluation of forecasting neural network models.

• [x] Implemented infrastructure for hyperparameter grid search.

    class ForecastModel(Model):
    	def forecast(self, x, y, batch_size=None, verbose=0, steps=None):
        def evaluate_forecast(self, x, y, batch_size=None, verbose=0, steps=None):
  
    class ModelWrapper(KerasRegressor):
        def forecast(self, x, y, **kwargs):
        def score_forecast(self, x, y, **kwargs):
  
  
    class GridSearch:
        def __init__(self, build_fn, param_grid, num_runs=20):
        def grid_search(self, x_train, y_train, x_val, y_val, **fit_params):
        def save_best(self, params, fit_params):