Parallel processing remote and server logs collection / scp

1 次查看（过去 30 天）

显示更早的评论

Life is Wonderful 2021-6-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/862020-parallel-processing-remote-and-server-logs-collection-scp

关闭： Life is Wonderful 2021-6-28

在 MATLAB Online 中打开

I would like to stream a file using HTTP endpoint / scp so that I can collect the logs from a remote server.

Is there a way ,parallel.internal.logging.enableClientLogging or parallel can help me connecting the input and output model. I have model where I use ugetdir to collect logs and then process the logs.

But now I have larger system where logs should be downloaded and processed.

I accept a HTTP POST request which will contain the following JSON in its body:

{ 
      event, 
      entity, 
      actor, 
      machines, 
      tests, 
      options 
} 

I'II call back the HTTP domain name mentioned in the actor, so that it will give back the JSON of all the log files generated for that job

Can you please suggest me how (

JobStorageLocation

) can do this ?

Thank you!

6 个评论
显示 4更早的评论隐藏 4更早的评论

Life is Wonderful 2021-6-23

编辑：Life is Wonderful 2021-6-28

在 MATLAB Online 中打开

Thanks @Edric Ellis ,

Probably I did not explain well,I'II try again

I want MATLAB to read test results produced by NodeJS ,

your best bet is to put the files there ?

Question and Help: I need matlab pseudo code using below attributes from Nodejs to put the files to JobStorageLocation and let workers pick them afterwards

At the moment, I am looking how Matlab can run HTTP server with the following endpoint.

Accept a HTTP POST request which will contain the following JSON in its body:

    {       event,       entity,       actor,       machines,       tests,       options       } 

I am interested in the event (‘completed’) and entity (‘job’).

Then I'II call back the HTTP domain name mentioned in the actor with the following call:

URL = actor + /api/logs/:id

Where id will be options.id that received in the previous callback.

This will give back the JSON of all the log files generated for that job.

I can then generate the full path for any log that I want:

URL = actor + logs/:id/:path

I can then process the log and create final log.

Once I am are ready,

I should do a HTTP post request to: URL = actor + logs + :id + :name

With Nodejs support , the file should be streamed via the body

Thanks a lot!

Edric Ellis 2021-6-23

在 MATLAB Online 中打开

I'm afraid I still don't understand where (i.e. on which host, caused by which process) the files are being generated. There is no portable way to store arbitrary files inside JobStorageLocation, the only documented API is to use properties of the Job and Task objects. If the data needs to be transferred from client to worker, you could read the files and put the resulting data structure into some property of the Job or Task; if the data needs to be transferred from worker to client, you could read the file(s) and return the value from your task's function. So, you could do something like this:

% Example sending JSON data to a worker for processing
jsonFile = 'somePathAtClient.json';
jsonData = jsondecode(fileread(jsonFile));
job = batch(@processJsonData, 1, {jsonData});
function out = processJsonData(jsonData)
    out = doStuff(jsonData);
end

Or, in the other direction:

job = batch(@returnJsonData, 1, {args..});
function out = returnJsonData(args..)
    % do stuff ...
    jsonFile = 'somePathOnTheWorker.json';
    jsonData = jsondecode(fileread(jsonFile));
    out = doStuff(jsonData);
end

I don't know if that helps.

Edric Ellis 2021-6-24

Sorry, I don't know anything about MATLAB Production Server or matlab.net.http. You might be better off either asking a separate question about that piece.

Life is Wonderful 2021-6-24