Commit d0c59879 authored by jvfpw18's avatar jvfpw18

Merge branch 'v1.1.0' into 'master'

v1.1.0

See merge request !13
parents 610839ff e1be7d28
# Changelog
All changes between versions will be documented in this file.
## 1.1.0 - 2019-10-15
### New Features
* Added integration of sql, python and bash scripts with the command `run_script`.
* Added group execution of sql scripts with the commands `execute_sql_group`, `drop_group` and `rebuild_group`.
The groups are defined in the configuration file hotmapper/database/`groups.py`.
* The run_aggregations command now also run all denormalizations (It executes any special condition `~` that doesn't need
the data CSV.).
* All columns information will also be contained in the `table_definition JSON`, it'll be updated automatically when a
remap is executed. This permits easier access to all the table information.
* You can now create a table only with it's table_definition. (It needs the columns parameter in the JSON,
you still won't be able to insert data without a mapping_protocol CSV.)
* You can now, create, delete and rename columns by editing directly the `table_definition JSON`. (The mapping_protocol
CSV won't be automatically updated and the insertion of data still requires the presence of the columns there.)
* Added a confirmation prompt when doing a remap.
### Fixes
* Fixed insertion error when the header of the data csv contains `"` (quotes).
* Fixed being unable to run denormalizations when the table didn't contain an `ano_censo` column. Now it will try to get
the column `YEAR_COLUMN` defined in the `settings.py` as condition a of the denormalization/aggregation query.
* Fixed insertion of data when two columns of the mapping_protocol where mapped directly to the same header from the data CSV.
* Fixed foreign key error with sqlalchemy-monetdb (dependency updated).
### Code changes
* Updated all dependencies, adjusted the code accordingly with possible changes.
* Added new functional tests. Can be run executing `python -m tests.database_test test_all`
* Refactored table_definition into a class, similar to protocol.
\ No newline at end of file
This diff is collapsed.
# HOTMapper #
This respository contains the HOTMapper tool, a tool that allows the user to manage his historical data using a mapping protocol. This specific version is freezed for demonstration purposes for the EDBT 2019 conference.
This respository contains the HOTMapper tool, a tool that allows the user to manage his historical data using a mapping protocol.
Bellow we have a simple usage tutorial, if you want a more complete tutorial or know more about all HOTMapper aspects,
please head to our [wiki page.](https://gitlab.c3sl.ufpr.br/tools/hotmapper/wikis/home)
## Table of content ##
......@@ -17,20 +20,20 @@ This respository contains the HOTMapper tool, a tool that allows the user to man
The Open Data sources extracted and processed by the tool can be found at the link: [INEP](http://portal.inep.gov.br/web/guest/microdados) in the section "Censo Escolar" and "Censo da Educação Superior".
To make it easier to execute the tool, we have dowloaded all data from "Local Oferta" is in the directory open_data. This way it is not necessary to search for the original sources.
To make it easier to execute the tool, we have dowloaded all data from "Local Oferta" is in the directory `open_data`. This way it is not necessary to search for the original sources.
**NOTE**: It's important to verify if there is a column identifying the year of the dataset;
**NOTE**: It's important to verify if there is a column identifying the year of the dataset
## Requirements ##
* Python 3 (It's recommended to use a virtual environment, such as virtualenv)
* MonetDB (We plan to make other databases to work with HOTMapper in the future)
* [MonetDB](https://www.monetdb.org/Downloads) (We plan to make other databases to work with HOTMapper in the future)
## Installation ##
----
**NOTICE:**
We assume thatPython 3.x is installed in the local computer and that all the following commands that use Python will use Python 3.x.
We assume that Python 3.x is installed in the local computer and that all the following commands that use Python will use Python 3.x.
--
1) Install virtualenv
......@@ -38,7 +41,7 @@ We assume thatPython 3.x is installed in the local computer and that all the fol
1a) On Linux/macOS
```bash
$ sudo -H pip install virtualenv
$ sudo -H pip3 install virtualenv
```
1b) On Windows (with administrator privilleges)
......@@ -52,6 +55,11 @@ $ pip install virtualenv
```bash
$ git clone git@gitlab.c3sl.ufpr.br:tools/hotmapper.git
```
or
```bash
$ git clone https://github.com/C3SL/hotmapper.git
```
3) Go to the repository
......@@ -85,7 +93,7 @@ $ .\env\Scripts/activate
$ pip install -r requirements.txt
```
## Command Line Interface ##
## Command Line Interface (CLI) ##
The CLI (Command Line Interface) uses the standard actions provided by manage.py, which means that to invoke a command it follows the following patterns:
......@@ -101,7 +109,7 @@ Where COMMAND can be:
$ python manage.py create <table_name>
```
**Notice** that the HOTMapper will use the name of the protocol as the name of the table.
**NOTICE** that the HOTMapper will use the name of the protocol as the name of the table.
* insert: Inserts a CSV file in an existing table.
......@@ -164,66 +172,66 @@ $ python manage.py generate_backup
```
## Demo scenarios ##
In this Section we will explain how to execute the demo scenarios that were submitted to EDBT 2019. Demo scenario 1 uses the dataset "local oferta", which is included in the directory open_data. Demo scenario 2 uses the dataset "matricula" which can be downloaded from the [INEP's Link ](http://portal.inep.gov.br/web/guest/microdados) in the section "Censo Escolar".
In this Section we will explain how to execute the demo scenarios that were submitted to EDBT 2019. Demo scenario 1 uses the dataset "local oferta", which is included in the directory `open_data`. Demo scenario 2 uses the dataset "matricula" which can be downloaded from the [INEP's Link ](http://portal.inep.gov.br/web/guest/microdados) in the section "Censo Escolar".
In both scnearios, we assume that you started the virtual environment as explained in Section `Installation - 5`
In both scenarios, we assume that you started the virtual environment as explained in Section `Installation - 5`
### Demo scenario 1 ###
This section contains the commands used in the scenario 1, which is the creation of a new data source and the inclusion of the corresponding data.
This Section contains the commands used in the scenario 1, which is the creation of a new table and the inclusion of the corresponding data.
1) First we need to create the database, to do so we execute the following command:
1) First we need to create the table in the database, to do so we execute the following command:
```bash
$ ./manage.py create localoferta_ens_superior
```
2) Now, as we already have the mapping definition, we need to insert the open data in the data base. To do it we must execute the following commands:
2) Now, as we already have the mapping definition, we need to insert the open data in the database. To do it we must execute the following commands:
**NOTE:** FILEPATH is the **_full path_** for the directory where the open data table is, for example (in a Linux environment): `/home/c3sl/HOTMapper/open_data/DM_LOCAL_OFERTA_2010`
**NOTE:** FILEPATH is the **_full path_** for the directory where the open data table is, for example (in a Linux environment): `/home/c3sl/HOTMapper/open_data/DM_LOCAL_OFERTA_2010.CSV`
a) To insert 2010:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2010.CSV localoferta_ens_superior 2010 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2010.CSV localoferta_ens_superior 2010 --sep="|"
```
b) To insert 2011:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2011.CSV localoferta_ens_superior 2011 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2011.CSV localoferta_ens_superior 2011 --sep="|"
```
c) To insert 2012:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2012.CSV localoferta_ens_superior 2012 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2012.CSV localoferta_ens_superior 2012 --sep="|"
```
d) To insert 2013:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2013.CSV localoferta_ens_superior 2013 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2013.CSV localoferta_ens_superior 2013 --sep="|"
```
e) To insert 2014:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2014.CSV localoferta_ens_superior 2014 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2014.CSV localoferta_ens_superior 2014 --sep="|"
```
f) To insert 2015:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2015.CSV localoferta_ens_superior 2015 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2015.CSV localoferta_ens_superior 2015 --sep="|"
```
g) To insert 2016:
```bash
$ ./manage.py insert FILEPATH/DM_LOCAL_OFERTA_2016.CSV localoferta_ens_superior 2016 --sep="|"
$ ./manage.py insert FILEPATH/hotmapper/open_data/DM_LOCAL_OFERTA_2016.CSV localoferta_ens_superior 2016 --sep="|"
```
### Demo scenario 2 ###
This section contains the commands used in the scenario 2, which is an update of an existing data source.
This Section contains the commands used in the scenario 2, which is an update of an existing table.
1) First we need to create the database, to do so execute the following command:
1) First we need to create the table in the database, to do so we execute the following command:
```bash
$ ./manage.py create matricula
```
......@@ -252,19 +260,19 @@ d) To insert 2016:
$ ./manage.py insert FILEPATH/MATRICULA_2016.CSV matricula 2016 --sep="|"
```
3) Change the matricula's mapping protocol. You can use the `matricula_remap.csv` (To do so, rename the current `matricula.csv` to something else and the `matricula_remap.csv` to `matricula.csv`). In that case, the only column that will change is the "profissionalizante", because now, instead of the ELSE returns 0 it returns 9.
3) Change the matricula's mapping protocol. You can use the `matricula_remap.csv` (To do so, rename the current `matricula.csv` to something else and the `matricula_remap.csv` to `matricula.csv`). In that case, the only column that will change is the "profissionalizante", because now, instead of the `ELSE returns 0` it returns `9`.
4) Run the remap command
```bash
$ ./manage.py remap matricula
```
The above command will update the table `Fonte` and the schema from the table matricula
The above command will update the table `Fonte` and the schema from the table `matricula`
5) Update the table
```bash
$ ./manage.py update_from_file /FILEPATH/MATRICULA_2013.CSV matricula 2013 --columns="profissionalizante" --sep="|"
$ ./manage.py update_from_file FILEPATH/MATRICULA_2013.CSV matricula 2013 --columns="profissionalizante" --sep="|"
```
The above command will update the data in the table matricula.
The above command will update the data in the table `matricula`.
#!/bin/bash
# ---------------------------------------------------------------------------------------#
# Esse script tem como objetivo facilitar a criação do banco de dados do projeto SIMCAQ,
# conforme a necessidade dos desenvolvedores. O código é livre para modificações contanto
# que os que utilizam o script sejam notificados das mudanças decorrentes.
# ---------------------------------------------------------------------------------------#
# ---------------------------------------------------------------------------------------#
# Função para criar as tabelas que são consideradas bases para o banco de dados
# ---------------------------------------------------------------------------------------#
fBase ()
{
mclient -d $1 base/regiao.sql
mclient -d $1 base/estado.sql
mclient -d $1 base/municipio.sql
mclient -d $1 base/siope_uf.sql
mclient -d $1 base/siope_mun.sql
mclient -d $1 base/siope_mun_seed.sql
mclient -d $1 base/instituicao_superior.sql
mclient -d $1 base/formacao_superior.sql
mclient -d $1 base/formacao_superior_seed.sql
mclient -d $1 base/ibge_pib.sql
}
# ---------------------------------------------------------------------------------------#
# ---------------------------------------------------------------------------------------#
# Função para criar as tabelas a partir dos protocolos de mapeamento
# ---------------------------------------------------------------------------------------#
fCreate ()
{
./manage.py create escola
./manage.py create turma
./manage.py create matricula
./manage.py create docente
}
# ---------------------------------------------------------------------------------------#
# ---------------------------------------------------------------------------------------#
# Função para inserir dados nas tabelas criadas a partir dos protocolos de mapeamento
# ---------------------------------------------------------------------------------------#
fInsert()
{
local alpha="$2"
while [ "$alpha" -le "$3" ]; do
./manage.py insert $1${alpha}_ESCOLAS.CSV escola $alpha --sep=\|
./manage.py insert $1${alpha}_TURMAS.CSV turma $alpha --sep=\|
./manage.py insert $1${alpha}_DOCENTES_CO.CSV docente $alpha --sep=\|
./manage.py insert $1${alpha}_DOCENTES_NORTE.CSV docente $alpha --sep=\|
./manage.py insert $1${alpha}_DOCENTES_NORDESTE.CSV docente $alpha --sep=\|
./manage.py insert $1${alpha}_DOCENTES_SUDESTE.CSV docente $alpha --sep=\|
./manage.py insert $1${alpha}_DOCENTES_SUL.CSV docente $alpha --sep=\|
./manage.py insert $1${alpha}_MATRICULA_CO.CSV matricula $alpha --sep=\|
./manage.py insert $1${alpha}_MATRICULA_NORTE.CSV matricula $alpha --sep=\|
./manage.py insert $1${alpha}_MATRICULA_NORDESTE.CSV matricula $alpha --sep=\|
./manage.py insert $1${alpha}_MATRICULA_SUDESTE.CSV matricula $alpha --sep=\|
./manage.py insert $1${alpha}_MATRICULA_SUL.CSV matricula $alpha --sep=\|
alpha=$(($alpha + 1))
done
}
# ---------------------------------------------------------------------------------------#
# ---------------------------------------------------------------------------------------#
# Retorna uma ajuda caso não haja parâmetros de entrada
# ---------------------------------------------------------------------------------------#
if [ ! $1 ]; then
printf "\n# WARNING: Don't forget to check the settings file for the database name.\n"
printf "\n# This script has 4 commands:\n"
printf "# 1. all: execute all commands to create the database and insert data.\n"
printf "# 2. base: execute the commands to create de base tables.\n"
printf "# 3. create: execute the commands to create the tables.\n"
printf "# 4. insert: execute the commands to insert data to tables.\n\n"
printf "# Estructure of commands:\n"
printf "# 1. ./auto.sh all [database_name] [path_to_files] [initial_year]"
printf " [final_year]\n"
printf "# 2. ./auto.sh base [database_name]\n"
printf "# 3. ./auto.sh create\n"
printf "# 4. ./auto.sh insert [path_to_files] [initial_year] [final_year]\n\n"
exit 0;
fi
# ---------------------------------------------------------------------------------------#
# ---------------------------------------------------------------------------------------#
# Execução do script conforme os comandos passados
# ---------------------------------------------------------------------------------------#
source ./env/bin/activate
if [ $? = 0 ]; then
printf "\n# Environment activated!\n"
if [ "$1" = 'all' ]; then
if [ $2 ] && [ $3 ] && [ $4 ] && [ $5 ]; then
printf "\n# Initializing the creation of base tables (may need database"
printf " password)...\n"
sleep 1
fBase "$2"
printf "\n# Initializing the creation of mapping tables...\n"
sleep 1
fCreate
printf "\n# Initializing the insertion of data, this may take a while...\n"
sleep 2
fInsert "$3" "$4" "$5"
sleep 1
else
printf "# ERROR: Missing parameters!\n"
exit -1;
fi
elif [ "$1" = 'base' ]; then
if [ $2 ]; then
printf "\n# Initializing the creation of base tables (may need database"
printf " password)...\n"
sleep 1
fBase "$2"
sleep 1
else
printf "# ERROR: Missing parameters!\n"
exit -1;
fi
elif [ "$1" = 'create' ]; then
printf "\n# Initializing the creation of tables...\n"
sleep 1
fCreate
sleep 1
elif [ "$1" = 'insert' ]; then
if [ $2 ] && [ $3 ] && [ $4 ]; then
printf "\n# Initializing the insertion of data, this may take a while...\n"
sleep 2
fInsert "$2" "$3" "$4"
sleep 1
else
printf "# ERROR: Missing parameters!\n"
exit -1;
fi
else
printf "\n# ERROR: Missing parameters!\n"
deactivate
printf "\n# Environment deactivated!\n"
printf "# Terminating...\n"
sleep 1
exit -1;
fi
deactivate
printf "\n# Environment deactivated!\n"
printf "\n# All done! Terminating...\n"
sleep 1
exit 0;
else
printf "# ERROR: can't find the directory for environment!\n"
exit -1;
fi
"""
Copyright (C) 2018 Centro de Computacao Cientifica e Software Livre
'''
Copyright (C) 2016 Centro de Computacao Cientifica e Software Livre
Departamento de Informatica - Universidade Federal do Parana - C3SL/UFPR
This file is part of HOTMapper.
......@@ -15,7 +15,6 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with simcaq-cdn. If not, see <https://www.gnu.org/licenses/>.
"""
along with HOTMapper. If not, see <https://www.gnu.org/licenses/>.
'''
"""
Copyright (C) 2018 Centro de Computacao Cientifica e Software Livre
'''
Copyright (C) 2016 Centro de Computacao Cientifica e Software Livre
Departamento de Informatica - Universidade Federal do Parana - C3SL/UFPR
This file is part of HOTMapper.
......@@ -15,23 +15,26 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with simcaq-cdn. If not, see <https://www.gnu.org/licenses/>.
"""
along with HOTMapper. If not, see <https://www.gnu.org/licenses/>.
'''
'''Database manipulation actions - these can be used as models for other modules.'''
import logging
from sqlalchemy import create_engine, MetaData
from sqlalchemy import create_engine, MetaData, text
from os import chdir
from datetime import datetime
from database.base import MissingTableError
from database.database_table import gen_data_table, copy_tabbed_to_csv
import database.groups
import settings
from database.groups import DATA_GROUP, DATABASE_TABLE_NAME
import pandas as pd
ENGINE = create_engine(settings.DATABASE_URI, echo=settings.ECHO)
META = MetaData(bind=ENGINE)
logging.basicConfig(format = settings.LOGGING_FORMAT)
logger = logging.getLogger(__name__)
database_table_logger = logging.getLogger('database.database_table')
database_table_logger.setLevel(settings.LOGGING_LEVEL)
......@@ -42,8 +45,8 @@ sqlalchemy_logger.setLevel(settings.LOGGING_LEVEL)
def temporary_data(connection, file_name, table, year, offset=2,
delimiters=[';', '\\n', '"'], null=''):
header = open(file_name, encoding="ISO-8859-9").readline().strip()
header = header.split(delimiters[0])
header = pd.read_csv(file_name, encoding="ISO-8859-9", sep=delimiters[0], nrows=1)
header = [h.strip() for h in header.columns.values]
ttable = table.get_temporary(header, year)
ttable.create(bind=connection)
......@@ -68,13 +71,14 @@ def insert(file_name, table, year, offset=2, delimiters=[';', '\\n', '"'], null=
trans.commit()
def create(table):
def create(table, ignore_definitions=False):
'''Creates table from mapping_protocol metadata'''
table = gen_data_table(table, META)
table.gen_definitions()
with ENGINE.connect() as connection:
trans = connection.begin()
table.create(bind=connection)
table.create(bind=connection, ignore_definitions=ignore_definitions)
table.set_source(bind=connection)
table.create_mapping_table(bind=connection)
trans.commit()
......@@ -85,12 +89,13 @@ def drop(table):
table.drop()
def remap(table):
def remap(table, auto_confirmation=True, verify_definitions=False):
'''Applies change made in mapping protocols to database'''
table = gen_data_table(table, META)
table.gen_definitions()
table.map_from_database()
table.remap()
table.remap(auto_confirmation, verify_definitions)
def csv_from_tabbed(table_name, input_file, output_file, year, sep=';'):
table = gen_data_table(table_name, META)
......@@ -101,7 +106,7 @@ def csv_from_tabbed(table_name, input_file, output_file, year, sep=';'):
copy_tabbed_to_csv(input_file, column_mappings, settings.CHUNK_SIZE, output_file,
column_names=column_names, sep=sep)
def update_from_file(file_name, table, year, columns=None, target_list=None,
def update_from_file(file_name, table, year, columns=None,
offset=2, delimiters=[';', '\\n', '"'], null=''):
'''Updates table columns from an input csv file'''
table = gen_data_table(table, META)
......@@ -110,7 +115,7 @@ def update_from_file(file_name, table, year, columns=None, target_list=None,
raise MissingTableError(table.name)
if columns is None:
columns = []
columns = [c.name for c in table.columns]
with ENGINE.connect() as connection:
trans = connection.begin()
......@@ -140,3 +145,37 @@ def generate_backup():
f = open(settings.BACKUP_FILE,"w")
f.write(str(datetime.now()))
f.close()
def execute_sql_script(sql_scripts, sql_path=settings.SCRIPTS_FOLDER):
if type(sql_scripts) == str:
sql_scripts = [sql_scripts]
with ENGINE.connect() as connection:
trans = connection.begin()
for script in sql_scripts:
with open(sql_path + '/' + script) as sql:
connection.execute(text(sql.read()))
trans.commit()
def execute_sql_group(script_group, sql_path=settings.SCRIPTS_FOLDER, files=False):
if not files:
sql_script = [DATA_GROUP[group.upper()] for group in script_group.split(",")]
else:
sql_script = script_group.split(",")
for sql in sql_script:
execute_sql_script(sql, sql_path + '/')
def drop_group(script_group, files=False):
script_group = script_group.split(",")
selected_tables = []
if not files:
for group in script_group:
selected_tables += DATA_GROUP[group.upper()]
else:
selected_tables = script_group
for table in reversed(selected_tables):
if table in DATABASE_TABLE_NAME:
table_name = DATABASE_TABLE_NAME[table]
else:
table_name = table.replace('.sql', '')
drop(table_name)
"""
Copyright (C) 2018 Centro de Computacao Cientifica e Software Livre
'''
Copyright (C) 2016 Centro de Computacao Cientifica e Software Livre
Departamento de Informatica - Universidade Federal do Parana - C3SL/UFPR
This file is part of HOTMapper.
......@@ -15,9 +15,8 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with simcaq-cdn. If not, see <https://www.gnu.org/licenses/>.
"""
along with HOTMapper. If not, see <https://www.gnu.org/licenses/>.
'''
'''Module containing base declarations'''
......@@ -42,6 +41,11 @@ class MissingProtocolError(DatabaseError):
requires a protocol while there is none loaded'''
pass
class MissingDefinitionsError(DatabaseError):
'''This exception should be raised if the program tries to use methods that
requires the table Definitions while there is none loaded'''
pass
class MissingForeignKeyError(DatabaseError):
'''This exception should be raised if an expected foreign key is not found.'''
def __init__(self, referred_table=None):
......
This diff is collapsed.
'''
Copyright (C) 2016 Centro de Computacao Cientifica e Software Livre
Departamento de Informatica - Universidade Federal do Parana - C3SL/UFPR
This file is part of HOTMapper.
HOTMapper is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
HOTMapper is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with HOTMapper. If not, see <https://www.gnu.org/licenses/>.
'''
import logging
import os
import json
import jsbeautifier
import settings
from database.base import InvalidTargetError
logger = logging.getLogger(__name__)
standard_keys = {
'source': 'data_source',
'description': 'pairing_description',
'pkcolumns': 'pk',
'fkcolumns': 'foreign_keys',
'columns': 'columns'
}
class Definitions(object):
'''
Class created from the Table definitions, contains primary key, foreign key, descriptions, source
and columns
'''
def __init__(self, t_name, keys=None):
self.source = None
self.description = None
self.columns = None
self.pkcolumns = None
self.fkcolumns = None
self._name = t_name
self.load_json(keys)
def load_json(self, keys=None):
''' Read the table definition json into the correct Definitions variables '''
definitions = self._name + '.json'
logger.debug("Acquiring definitions from %s", definitions)
definitions = os.path.join(settings.TABLE_DEFINITIONS_FOLDER, definitions)
definitions = json.loads(open(definitions).read())
self.load_from_dict(definitions, keys)
def update_columns(self, columns):
''' Update Table definition json with a new columns dict '''
definitions_json = self._name + '.json'
logger.debug("Updating table definitions from %s", definitions_json)
definitions_json = os.path.join(settings.TABLE_DEFINITIONS_FOLDER, definitions_json)
self.columns = columns
new_definitions = self.to_dict()
new_definitions = jsbeautifier.beautify(json.dumps(new_definitions, ensure_ascii=False))
with open(definitions_json, "w") as def_json:
def_json.write(new_definitions)
logger.debug("Definitions Updated")
def load_from_dict(self, definitions, keys=None):
''' Takes a definitions dictionary and load the object Definitions variables '''
if not keys:
keys = standard_keys
self.source = definitions[keys['source']]
self.description = definitions[keys['description']]
self.pkcolumns = definitions[keys['pkcolumns']]
self.fkcolumns = definitions[keys['fkcolumns']]
try:
self.columns = definitions[keys['columns']]
except KeyError:
self.columns = None
logger.debug("Definitions loaded")
def to_dict(self, keys=None):
''' Transforms a Definition object into a dictionary for writing in a json file '''
if not keys:
keys = standard_keys
definitions = {
keys['description']: self.description,
keys['source']: self.source,
keys['pkcolumns']: self.pkcolumns,
keys['fkcolumns']: self.fkcolumns,
keys['columns']: self.columns
}
return definitions
def get_targets(self):
''' Returns a list containing all columns targets '''
targets = []
for column_name, parameter_list in self.columns.items():
targets.append(parameter_list[1])
return targets
def get_dbcolumn_from_target(self, target):
''' Gets a database column from a target column name. Ouput is a list
with the column name and type contents.
:return: ['column_name','column_type'] '''
found = False
for column_name, parameter_list in self.columns.items():
if parameter_list[1] == target:
found = True
return [column_name, parameter_list[0]]
if not found:
raise InvalidTargetError(target)
'''
Copyright (C) 2016 Centro de Computacao Cientifica e Software Livre
Departamento de Informatica - Universidade Federal do Parana - C3SL/UFPR
This file is part of HOTMapper.
HOTMapper is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
HOTMapper is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with HOTMapper. If not, see <https://www.gnu.org/licenses/>.
'''
'''Group Settings'''
# ---------------------------------------------------------------------------------------#
# SMPPIR
# ---------------------------------------------------------------------------------------#
INEP = [
'admission.sql',
'course.sql',
'evader.sql',
'extracurricular_activities.sql',
'graduate.sql',
'institution.sql',
'institutionPrivate.sql',
'social_support.sql',
'student_loans.sql'
]
PROUNI = [
'coursePROUNI.sql',
'institutionPROUNI.sql',
'prouni.sql'
]
PNAD = [
'pnad.sql'
]
CADUNICO = [
'eixo2.sql',
'eixo3.sql',
'eixo4.sql',
'african_sustentability.sql',
'african_rights.sql',
'african_culture.sql'
]
FIES = [
'courseFIES.sql',
'fies.sql',
'institutionFIES.sql'
]
ALL_GROUPS_SMPPIR = INEP + PROUNI + PNAD + CADUNICO + FIES
# ---------------------------------------------------------------------------------------#
# ---------------------------------------------------------------------------------------#
# SIMCAQ
# ---------------------------------------------------------------------------------------#
BASE = [
'regiao.sql',
'estado.sql',
'municipio.sql',
'siope_uf.sql',
'siope_mun.sql',
'siope_mun_seed.sql',
'instituicao_superior.sql',
'formacao_superior.sql',
'formacao_superior_seed.sql',
'ibge_pib.sql',
'cub.sql',