Commit 8238998a authored by Cristian Weiland's avatar Cristian Weiland

Merge branch 'issue/76' into 'develop'

Issue #76: changing the name of some directories to more intuitive ones

See merge request !38
parents bbcd203e 046c423a
...@@ -13,4 +13,4 @@ O projeto contém dados referentes a: ...@@ -13,4 +13,4 @@ O projeto contém dados referentes a:
   ├── workers - Contém scripts para gerenciar dados de Servidores.    ├── workers - Contém scripts para gerenciar dados de Servidores.
   ├── travel_allowance - Contém scripts para gerenciar dados de Diárias de Viagens.    ├── travel_allowance - Contém scripts para gerenciar dados de Diárias de Viagens.
   ├── expenses - Contém scripts para gerenciar dados de Pagamentos.    ├── expenses - Contém scripts para gerenciar dados de Pagamentos.
   └── curl - Contém scripts que auxiliam na utilização do curl.    └── misc - Contém scripts que auxiliam na utilização do curl.
# Input: The name of a file containing a curl query.
# Output: The output for the curl query.
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <query-file>"
exit
fi
query=$(cat $1)
echo curl -u cw14:123mudar -XGET node1.c3sl.ufpr.br:9200/ufpr-servidores-*/_search?pretty -d \'"$query"\'
...@@ -5,8 +5,8 @@ ...@@ -5,8 +5,8 @@
# Input: Year, month and day from a CSV file, username and password. # Input: Year, month and day from a CSV file, username and password.
# Ex (inserting data from file 20130930_Cadastro.csv): ./create_config.py 2013 09 30 myuser mypassword # Ex (inserting data from file 20130930_Cadastro.csv): ./create_config.py 2013 09 30 myuser mypassword
# Output: This script will create two config files: # Output: This script will create two config files:
# - JSON: This config will be used for script merge_files_es.py, and will be stored in transparencia/configs/workers/JSON, with its name being config-year-month. # - JSON: This config will be used for script merge_files.py, and will be stored in transparencia/configs/workers/JSON, with its name being config-year-month.
# - Logstash: This config will be used by logstash to insert the resulting CSV from merge_files_es.py into ElasticSearch. # - Logstash: This config will be used by logstash to insert the resulting CSV from merge_files.py into ElasticSearch.
import sys, csv, json, math, subprocess import sys, csv, json, math, subprocess
from pathlib import Path from pathlib import Path
......
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
# This scripts does 4 things: # This scripts does 4 things:
# 1- Download required files and store them in the right place. # 1- Download required files and store them in the right place.
# 2- Create config files via create_config.py # 2- Create config files via create_config.py
# 3- Merge CSV data and create a new CSV file via merge_files_es.py. # 3- Merge CSV data and create a new CSV file via merge_files.py.
# 4- Insert CSV file generated in step 2 into ElasticSearch via Logstash. # 4- Insert CSV file generated in step 2 into ElasticSearch via Logstash.
# Input: Year, Month from CSV file, ElasticSearch's user and password. # Input: Year, Month from CSV file, ElasticSearch's user and password.
...@@ -109,7 +109,7 @@ do ...@@ -109,7 +109,7 @@ do
# Step 3: # Step 3:
# Start processing # Start processing
strReplacement=$( echo "${filter[$key]}" | sed 's/ /\\ /g' ) strReplacement=$( echo "${filter[$key]}" | sed 's/ /\\ /g' )
./merge_files_es.py $path/config-${year}-${month}.json "$strReplacement" "${columnName}" ./merge_files.py $path/config-${year}-${month}.json "$strReplacement" "${columnName}"
rm $path/${year}${month}${day}_Cadastro_Unique.csv rm $path/${year}${month}${day}_Cadastro_Unique.csv
# Step 4: # Step 4:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment